Data center intelligent control and optimization

ABSTRACT

Systems and methods of monitoring, analyzing, optimizing and controlling data centers and data center operations are disclosed. The system includes data collection and storage hardware and software for harvesting operational data from data center assets and operations. Intelligent analysis and optimization software enables identification of optimization and/or control actions. Control software and hardware enables enacting a change in the operational state of data centers.

RELATED APPLICATIONS

This application is a continuation-in-part and claims priority to andthe benefit of U.S. patent application Ser. No. 13/788,834 filed Mar. 7,2013 and entitled “Data Center Intelligent Control and Optimization.”The '384 application claims priority to U.S. patent application Ser. No.12/626,299 filed Nov. 25, 2009 and entitled “Apparatus and Method ofEnvironmental Condition Management for Electronic Equipment.” The '299application claims priority to U.S. Provisional Patent Application No.61/119,980 filed Dec. 4, 2008 entitled, “Thermal Management Cabinet forElectronic Equipment.” The '299 and the '980 applications areincorporated by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to data centers, and moreparticularly to monitoring, optimizing and controlling data centeroperations.

BACKGROUND

Data centers are typically facilities for housing electronic equipment.Traditionally, these facilities often have a large footprint due to thevarious components necessary for maintaining the facilities, includingcooling equipment. Monitoring, optimizing and controlling a data centerand/or data center infrastructure assets is typically performed manuallywithout considering multiple aspects of the data center. For example,managing the data center infrastructure (e.g., cooling, powerdistribution, etc.) is typically done without considering theutilization of servers within the data center. Similarly, management ofthe servers (and application running on the servers) is typicallyignorant of the underlying data center infrastructure.

SUMMARY OF THE INVENTION

Embodiments of this disclosure include systems and methods of optimizingdata centers and data center operations by: collecting, monitoring,analyzing, warehousing and mining data; analyzing and predicting usingproprietary algorithms, forecasts, simulations, and models to developalternative data center configurations and processes; optimizing byanalyzing a plurality of optimization dimensions and developing andevaluating alternative optimization actions; and generating andimplementing optimization instructions.

In various embodiments, a data center control system, comprises a memoryin communication with a processor, the processor when executing acomputer program for data center control, performs operations tooptimize data center operations. The system accesses input dataassociated with data center operations, wherein the input data comprisesat least one of inventory data, operational data, historical data andexternal data. The system determines at least one data centeroptimization dimension. Optimization dimensions may include at least oneof reliability, resiliency, risk, latency, efficiency, cost and threatavoidance.

Based upon at least a portion of the input data the system forecasts toproduce predictive data. Based upon at least one data centeroptimization dimension, the predictive data and the input data, thesystem determines an optimization action. The system generates anoptimization instruction based upon the optimization action. In variousembodiments, executing the optimization instruction causes a change inan operational state associated with the data center operations.

Various embodiments of this disclosure include systems and methods ofmonitoring, analyzing and controlling data centers and related hardwareand software. In an embodiment, a data center infrastructure controlsystem (“DCICS”) determines, based upon a first expression, a firstoperation and data collection points associated with data center assets.Data collection points may include sensors, indicators, detectors,application programming interfaces, system data, etc. In variousembodiments, the expression may be obtained from a data base, a userinterface, another system, a hardware device, etc. In variousembodiments, the expression may be derived or generated by the DCICSbased upon, for example, other expressions, data or events.

DCICS interprets and/or analyzes the expression and generates machinecode instructions that may include, for example, instructions to obtainthe first point data from the data collection points and to perform theoperation based upon the first point data. In various embodiments, theinstructions may include instructions for obtaining, receiving orretrieving the first point data.

DCICS executes the machine code instructions to produce a first resultof the first operation. In various embodiments, in response to producingthe first result, DCICS examines or analyzes the first result anddetermines a first control instruction. Executing the first controlinstruction causes a change in an operational state of at least one of adata center asset and a data center. In various embodiments, DCICS mayexecute the control instruction, partially execute the controlinstruction, and/or communicate the control instruction to a hardwaredevice or software program for execution.

Other aspects and advantages of the present invention will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, illustrating the principles of theinvention by way of example only.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of the presentinvention, as well as the invention itself, will be more fullyunderstood from the following description of various embodiments, whenread together with the accompanying drawings, in which:

FIG. 1 depicts logical data center operations data sources leveraged bythe technology, in accordance with various embodiments;

FIG. 2 illustrates a data center, in accordance with variousembodiments;

FIG. 3 depicts a cross-section view of a data center module, inaccordance with various embodiments;

FIG. 4 depicts a data center module controller, in accordance withvarious embodiments;

FIG. 5 depicts a logical view of a data center management system, inaccordance with various embodiments;

FIG. 6 depicts a control system for a data center module, in accordancewith various embodiments;

FIG. 7 is a flow chart depicting a method of operating a data centermodule, in accordance with various embodiments; and

FIG. 8 depicts a modular data center, in accordance with variousembodiments.

DETAILED DESCRIPTION

Described herein is technology for data center monitoring, optimizationand control that leverages data from multiple components of the datacenter, data center related hardware and external data sources. Forexample, as described herein, a DCICS can communicate with multiplecomponents of the data center and/or external data sources to collectdata relating to and/or affecting the operation of the data center (datacenter operations data). The DCICS can modify the operation oroperational state of the data center based on the collected data centeroperations data.

While the description references specific technologies, systemarchitectures, operational control and data management techniques,practitioners will appreciate that this description is but variousembodiments and that other devices and/or methods may be implementedwithout departing from the scope of the invention. While the embodimentsdescribed herein are described in sufficient detail to enable thoseskilled in the art to practice the invention, it should be understoodthat other embodiments may be realized and that logical and mechanicalchanges may be made without departing from the spirit and scope of theinvention. Thus, the detailed description herein is presented forpurposes of illustration only and not of limitation.

For the sake of brevity, conventional data networking, applicationdevelopment and other functional aspects of the systems (and componentsof the individual operating components of the systems) may not bedescribed in detail herein. Furthermore, the connecting lines shown inthe various figures are intended to represent functional relationshipsand/or physical couplings between the various elements. It should benoted that many alternative or additional functional relationships orphysical connections may be present in various embodiments.

The benefits provided by this disclosure and technology include, forexample, increased efficiency of data center operations, reduced risk,lower costs, increased hardware and power utilization, increasedreliability, increased profitability, increased security, increasedresiliency, etc. For example, by leveraging the collected data centeroperations data, data center monitoring can be improved, the data centercan operate at a higher efficiency, and the amount and impact failureswithin the data center can be reduced.

The technology described herein can utilize a DCICS to collect datacenter operations data and/or control the operation of the data center.For example, the DCICS can communicate with and collect data fromvarious components of the data center and systems or “layers” associatedwith an information technology (“IT”) stack, including: data centerinfrastructure (e.g., utilities, environmental management equipment,power generation and distribution equipment, power backup equipment,data center modules, data center facilities, etc.); IT equipment in thedata center (e.g., PCs, servers, storage devices, networking equipment,etc.); software executing on IT equipment in, or associated with, thedata center (e.g., virtualization systems, operating systems, userapplications, etc.); and/or users in communication with any of theabove. In various embodiments, the technology can collect data fromexternal data sources, such as sources for weather data, seismic data,political data, economic (e.g., energy price changes), financial and/ornews data. DCICS can consolidate, aggregate, analyze, process, and/ordistill the collected information for analysis, storage and/orpresentation to data center operators.

In various embodiments, DCICS can analyze the collected data for itsimpact on data center operations and performance. For example, thecollected data can be analyzed in approximately real time to assesscurrent data center operations and/or forecast future data centeroperations, operational conditions, constraints, needs, costs,capabilities, preferred or optimal configurations, etc. DCICS can storeand analyze collected information to determine why certain eventsassociated with data center operations occurred and various DCICS canexecute various decision and control engines to develop and evaluatealternative data center configurations and implement changes in theoperational configuration or operational state of data centers.

FIG. 1 depicts logical data center data sources leveraged by thetechnology. In various embodiments, DCICS 105 can communicate with andenable transparency into all layers of the IT stack, including, forexample, a user layer, a virtualization layer, an application layer, anoperating system layer, an IT or hardware layer and a data centerinfrastructure layer. In the illustrated embodiment, DCICS 105 isbi-directionally connected (e.g., via a data connection) to translator107, data center infrastructure 110, IT hardware 115, virtualizationlayer 120, operating system 125, application 130, and/or user 135. Whileeach of the illustrated translator 107, data center infrastructure 110,IT hardware 115, virtualization layer 120, operating system 125,application 130, and user 135 are represented by a box in the logicaldiagram, it should be appreciated that each can represent multipleinstances of data center infrastructure, IT hardware, virtualizationlayers, operating systems, applications, and/or user.

Data center infrastructure 110 can include, for example, utilities,environmental management equipment, power generation and distributionequipment, power backup equipment, data center modules, and data centerfacilities. DCICS 105 can collect data center operations data from datacenter modules within a modular data center. For example, DCICS 105 cancollect data from one or more modular data centers and/or data centermodules, as described in U.S. patent application Ser. No. 13/751,568filed Jan. 28, 2013 and entitled “Modular Data Center,” the content ofwhich is hereby incorporated in its entirety. In various embodiments,DCICS 105 can collect data center operations data from traditional datacenters, such as raised floor facilities. DCICS 105 can sendinstructions (e.g., via data communications, control signal, etc.) todata center infrastructure 110. For example, DCICS 105 can cause thecooling equipment within a data center (or data center module) toincrease/lower the temperature, humidity, etc. in the data center (ordata center module).

In various embodiments, DCICS 105 may communicate with hardware elements(e.g., data center infrastructure 110, hardware 115, etc.) viatranslator 107. Translator 107 may be implemented as special purposehardware, software or a combination of hardware and software elements.In various embodiments translator 107 enables 150+ vendor protocollanguages, such as BACNet, Modbus, and Ethernet/IP, over a variety ofmedia types, such as Ethernet and RS-232/485, to communicate with DCICS105. Translator 107 is specially configured, custom designed and builthardware that “speaks” vendor protocol languages and then “translates”them into one language (e.g., OPC) that is “understood” by DCICS 105.

The translation capability of translator 107 enables enterprise-widecollection, storage, and display functions for a wide variety of datacenter infrastructure equipment, including building management, powerdistribution, and IT systems. Communications management is centralizedand simplified into a single platform accessible by DCICS 105. Invarious embodiments, translator 107 can easily be expanded throughplug-and-play device drivers, client interfaces, and hardware units. Invarious hardware embodiments of the translator, there is abundant roominside the translator enclosure for adding or removing snap-incomponents allowing translator 107 to be customized for virtually anyapplication.

IT hardware 115 can include, for example, PCs, servers, storage devices,networking equipment, and other computing devices located in one or moredata centers. For example, in various embodiments DCICS 105 can collector obtain temperature data and/or utilization data from, for example,one or more processor cores within a server, bandwidth utilization datafrom one or more network switches or routers, storage capacityinformation from one or more storage devices, etc. DCICS 105 can sendcontrol instructions and/or signals to IT hardware 115; for example,DCICS 105 can cause a server in the data center to power cycle,increase/reduce internal fan speed, etc.

Virtualization layer 120 can include, for example, virtualizationsoftware for managing usage and provisioning of IT hardware (e.g.,instances of VMWARE® software running on IT hardware 115). DCICS 105 cancollect data about the status, hardware allocation, and hardwareutilization of virtual machines resident on one or more servers. DCICS105 can send instructions to virtualization layer 120, for example, viaan API. For example, DCICS 105 can cause an instance of virtualizationsoftware to move a virtual machine to another physical server at adifferent geographic location.

Operating system 125 can include, for example, instances of MICROSOFTWINDOWS, LINUX, MAC OS X, UNIX etc. running on IT hardware 115 (nativelyor on virtual machines and/or partitions). For example, DCICS 105 cancollect data about the status of the operating system, such as systemload and which applications are running and/or loaded in memory. DCICS105 can control data center hardware and software; for example, bysending instructions to operating system 125. In various embodiments,DCICS 105 can cause the operating system instance to execute and/orterminate an application.

Application 130 can include instances of applications (e.g., web server,enterprise management, database, payroll, supply chain, etc. software)running on IT hardware 115. For example, DCICS 105 can collect dataabout an application's activity level, execution schedule, andanticipated duration. DCICS 105 can send instructions to application130. For example, DCICS 105 can initiate application 130, cause anapplication to execute a job, another application, a procedure, a datacommunication, etc.

User 135 can include applications and/or devices communicating with IThardware 115 located in the one more data centers managed by DCICS 105.For example, DCICS 105 can collect data from user 135 about distributingworkloads across IT hardware 115. DCICS 105 can send instructions touser 135.

Beneficially, by collecting data from and issuing instructions to datacenter infrastructure 110, IT hardware 115, virtualization layer 120,operating system 125, application 130, and user 135, DCICS 105 can moreefficiently and completely monitor, analyze, manage, predict, decide,optimize and control the operation of data centers. For example, theefficiency of application 130 can be increased by utilizing data centeroperations data from data center infrastructure 110.

FIG. 2 illustrates data center 200. Data center 200 can be composed ofmultiple data centers. As illustrated, data center 200 can include datacenter 210, data center 215, and data center 220. Data center 210 anddata center 215 can be modular data centers. Data center 210 and datacenter 215 can include multiple data center modules 225. Data centermodules 225 can contain IT equipment (not shown). Data center modules225 can, for example, be modules as described in U.S. patent applicationSer. No. 13/751,568 filed Jan. 28, 2013 and entitled “Modular DataCenter.” Data center 220 can be a traditional raised-floor data center,as is well known in the art. Data center 220 can contain IT equipment inrows 230.

Data center 200 can include resources in the same building. For example,data center 210, data center 215, and data center 220 can be housed inthe same building. Data center 200 can include resources ingeographically diverse locations. For example, data center 210 can be inthe United States, data center 215 can be in Brazil, and data center 220can be in Singapore.

Beneficially, the technology described herein can facilitate managementand control of heterogeneous data centers and other IT, datacommunications, power distribution and climate control resources. Forexample, in various embodiments, DCICS 105 manages, coordinates andcontrols data centers including a mixture of traditional (e.g.,“raised-floor”) data centers and modular data centers that may begeographically diverse and include heterogeneous and/or non-standardconfigurations of data center infrastructure and IT assets.

Referring now to FIG. 3, data center module 300 depicts a cross-sectionview of an embodiment of a data center module. One or more of modules225 can be structured and/or operate as described with respect to datacenter module 300. Data center module 300 contains floor 301 thatdivides the interior of data center module 300 into above-floor spaceand sub-floor space. Data center module 300 can contain IT equipment305. In the illustrated embodiment, data center module 300 contains airhandler unit (AHU) 310, which is supplied chilled fluid by chilled fluidsupply pipe 315 and fluid return pipe 320. In various embodiments,chilled fluid supply pipe 315 is connected to AHU 310 via a modulatedvalve. In various embodiments, the amount the modulated valve is opencan be controlled via control instructions from DCICS 105. Data centermodule 300 contains flexible barrier 325.

IT equipment 305 can be in fluid communication with pressurized plenum330. Pressurized plenum 330 can have one or more pressure sources, suchas AHU 310. AHU 310 can include a variable-speed,variable-frequency-drive fan. AHU 310 can be in communication withpressurized plenum 330 and configured to increase the pressure withinpressurized plenum 330. For example, in response to receiving aninstruction from DCICS 105 AHU 310 may activate its fan. IT equipment305 can separate the above-floor space into cold aisle 332 and hot aisle335. In the illustrated embodiment, cold aisle 332 and hot aisle 335 canprovide technicians access to the IT equipment 305. Flexible barrier 325can facilitate separation of cold aisle 332 and hot aisle 335 (aloneand/or in conjunction with IT equipment 305). It will be recognized thatthe use of the term “aisle” is suggestive of some embodiments and thatthe spaces or areas described as cold aisle 332 and hot aisle 335 can beof any shape, configuration or size.

AHU 310 can increase the pressure within pressurized plenum 330 togenerate a pressure differential between cold aisle 332 and hot aisle335, causing air 340 to flow across and/or through IT equipment 305. Theflow of air 340 across and/or through IT equipment 305 can remove heatfrom IT equipment 305, resulting in heated air 345. AHU 310, by use of,for example, a heat exchanger, can remove the heat from heated air 345.In various embodiments, AHU 310 utilizes a vapor-compression cycle heatexchanger. AHU 310 can transfer the heat to chilled fluid from chilledfluid supply pipe 315 and expel the heated fluid via fluid return pipe320.

Data center module 300 can include temperature and/or humidity sensors350 and 352, which can, for example, measure the temperature and/orhumidity of air in cold aisle 332 and hot aisle 335, respectively. Datacenter module 300 can include AHU output temperature and/or humiditysensor 355, which can measure the temperature and/or the humidity of airleaving AHU 310. Data center module 300 can include pressure sensors 360for measuring the differential pressure between cold aisle 332 and hotaisle 335. Data center module 300 can contain a power distribution unit(PDU) (not shown in FIG. 3). A PDU can distribute power to AHU 310, ITequipment 305, and other electrical equipment in data center module 300.A PDU can, for example, measure the power consumption of AHU 310 or ITequipment 305 individually and provide this data to DCICS 105.

FIG. 4 depicts data center module controller 400. Data center modulecontroller 400 can be housed in data center module 300. In variousembodiments, controller 400 comprises special-purpose hardware andsoftware. Data center module controller 400 can collect data relating tothe operation of data center module 300 and issue instructions to datacenter module 300 hardware. For example, data center module controller400 can obtain (e.g., retrieve or receive) temperature sensor data(e.g., from temperature and/or humidity sensors 350 and 352 and AHUoutput temperature and/or humidity sensor 355). Data center modulecontroller 400 can collect humidity sensor data (e.g., from temperatureand/or humidity sensors 350 and 352 and AHU output temperature and/orhumidity sensor 355). Data center module controller 400 can collect AHUdata 420 (e.g., data relating to the operation of AHUs, such asoperational status, fan speed, error status, power draw, etc.). Datacenter module controller 400 can collect power data 425 (e.g., powerdrawn by data center module 300 as a whole or by individual componentswithin data center module 300, such as AHU 310 or IT equipment 305).Data center module controller 400 can collect chilled fluid valve datafrom CCV 430 (e.g., position data, error status, etc.). Data centermodule controller 400 can collect IT equipment data 435 (e.g., datarelating to the operation of IT equipment in data center module 300,such as CPU utilization, CPU temperature, planar temperature, etc.).Data center module controller 400 can collect application data 440(e.g., data relating to the operation of applications running on ITequipment 305 in data center module 300). In various embodiments, datacenter module controller 400 can store the collected data. In variousembodiments, data center module controller 400 can provide the collecteddata to a data center management and control system, as described withrespect to FIG. 5. In various embodiments, controller 400 is configuredwith software modules (e.g., object code or machine readableinstructions) that enable receipt of control instruction from DCICS 105and execution of the instruction to control a one or more of ITequipment 305.

In various embodiments, data center module controller 400 integratessafety, motion, discrete, and drive capabilities in a single controller.data center module controller 400 may include compact I/O modules. Invarious embodiments, data center module controller 400 include: built-inRS-232 serial ports; built-in communication ports for EtherNet/IP orControlNet networks; a communication interface module for I/O controland remote device configuration; compact I/O modules providing acompact, DIN-rail or panel mounted I/O system. Software modules orcompiled machine language instructions may be developed, compiled andinstalled on data center module 400. In various embodiments, data centermodule 400 enables relay ladder, structured text, function blockdiagram, and sequential function chart editors to develop applicationprograms which can be downloaded to data center controller 400 memory.For example, programs may send control instuctions to module hardwareand software components, measure outside air, humidity, damperpositions, air flow rate of the fans, and more for the various equipmentand device sensors wired to the I/O modules. In various embodiments,data center module controller 400 exchanges information with DCICS 105in real-time.

FIG. 5 depicts a logical view of DCICS 105. CRM database 505 can be aCustomer Relationship Management database. Sales proposals, quotes,and/or customer information can be stored on CRM 505. Accountingdatabase 507 can be a system for financial reporting. Central database510 can be a repository for any data related to the operation of thedata centers managed by DCICS 105. Incident ticketing database 512 canbe software that facilitates customer ticketing and incident reporting.

Business intelligence and data warehouse database 515 can include anydata related to the operation of the data centers managed, tracked,controlled or stored by DCICS 105. In various embodiments, businessintelligence and data warehouse database 515 can implementcomputer-based techniques used in identifying, extracting, and enableanalyzing business data, such as sales revenue by products and/ordepartments, and/or by associated costs and incomes. Businessintelligence and data warehouse database 515 can provide historical,current and predictive views of business operations. For example,business intelligence and data warehouse database 515 can performreporting, online analytical processing, analytics, data mining, processmining, complex event processing, business performance management,benchmarking, text mining, forecasting and predictive analytics.

Central Data Manger 517 can access data from central database 510 andpresent it to a user (e.g., user 135 via a user interface of a clientdevice, an application programming interface (“API”), etc.). Dataquality monitor 520 can check data to verify that transactions areoccurring and validate that the transactions are producing the expectedresults. For example, data quality monitor 520 can sample the databasetables in central database 510 periodically to ensure that centraldatabase 510 is operating. In various embodiments, data quality monitorchecks for data central database 510 that violates business rules anddata quality monitor 520 creates and sends an email report on datavalidation issues.

Network monitor 522 can communicate with IT equipment 545 to determine,for example, the status of IT equipment 545. In various embodiments,network monitor 522 is an extensible tool that brings measurements fromnetworked devices into DCICS 105 where they can be stored and viewed. Invarious embodiments, network monitor 522 can monitor, for example:devices that support SNMP (simple network monitoring protocol); VMWare®virtual machines; EMU) storage arrays; other types of devices utilizingMicrosoft Windows PowerShell.

Notification processor 525 can generate notifications based on theoccurrence of an event, data, forecast, etc. In various embodiments,notifications may be system notifications, emails, text message,automated phone calls, tweets, operating system level messages (e.g. iOSnotification), etc.

Portal web services 527 can manage web-based customer interactions. Invarious embodiments, data from portal web services 527 can be directedto customers and/or data center management personnel (e.g., foraddressing customer issues). Interface services 530 can provideweb-based user interfaces.

Open Process Control (OPC) Server 537 is a software application thatacts as an API or protocol converter. In various embodiments, OPC Server537 can connect to a device such as, for exmple, a programmer logiccontroller, distributed control system, remote terminal unit, and/or ora data source such as a database or user interface, and translate thedata into a standards-based OPC format.

OPC client 532 can provide a logger to process real-time data from OPCserver 537. In various embodiments, OPC client 532 can permit parallelwrites as well as priority driven queuing of writes. OPC server 537 canreceive data center operations data from controller 535 (e.g., acontroller located in, or operatively connected to, a data centermodule, such as data center module controller 400). While theillustrated embodiment utilizes OPC, data can be exchanged in anyformat. Connectivity server 542 can facilitate access to controller 535.OPC converter 540 can convert data from connectivity server 542 to OPCformat, as necessary.

DCICS 105 can manage and control various types of data centers. DCICS105 can manage standalone data center 550. Standalone data center 550can include one or more data center modules. Standalone data center 550can use module database 552 to store collected data center operationsdata for standalone data center 550. In various embodiments, in theevent of a network outage between standalone data center 550 and centraldatabase 510, module database 552 can be used to monitor and/or controlstandalone data center 550. Replication server 555 can maintainsynchronization between module database 552 and central database 510.

Centralized data center 560 can operate in a manner similar tostandalone data center 550. In various embodiments such as the onedepicted in FIG. 5, centralized data center 560 stores data to a centraldatabase (e.g., central DB 510). In various embodiments, in the event ofa network outage between centralized data center 560 and centraldatabase 510, data relating to the operation of centralized data center560 may not be available centrally but the data relating to theoperation of centralized data center 560 can be buffered and provided tocentral database 510 when the connection is restored.

Traditional data center 570 may be, for instance, a raised floor datacenter. In various embodiments, traditional data center 570 can includesensors (e.g., temperature, humidity, pressure, motion detection, accesscontrol, acoustic, biometric, etc.). Sensor data can be collected by OPCserver 537. In various embodiments, OPC server 537 can communicate withIT equipment located in traditional data center 570. In variousembodiments, OPC server may exchange data with translator 107 and/ordata center module controller 400 (e.g. CTL 535 in FIG. 5).

In various embodiments, DCICS 105 can track components (e.g., logicaland physical hardware, software, systems) of data centers. In variousembodiments, DCICS 105 can group components of the data centers intopartitions. The term partition includes any logical or physical groupingof one or data center assets (hardware or software). For example, apartition can be defined to include a single server, multiple datacenter modules and everything contained therein, and/or a partition canbe defined to include an entire data center or group of data centers. Invarious embodiments, a partition may be associated with individual datacenter assets, individual sensors, etc.

Task engine 533 monitors changes to data collection points. In variousembodiments, in order to monitor changes to data, task engine 533 maypoll the data collection points, poll a database (e.g. central DB 510),receive data from a database (e.g via a database trigger), receivechanges made by a user, etc. In various embodiments, task engine 533detects a change made by a user to threshold or “setpoint” informationassociated with one or more data collection points. In an embodiment,task engine 533 includes an extender that enables modular plug-ins to beadded to a task manager (e.g. of a computer's operating system) so thatDCICS 105 can perform tasks, such as commanding an application, systemor computing environment (e.g., VMware environment) to gracefully shutdown or start up a virtual machine. In various embodiments, task engine533 associates data collection points to a shell command utility of anoperating system (e.g. Windows PowerShell commands), allowing commandsto be executed in response to users changing the setpoints (e.g.thresholds) data collection point from within DCICS 105. EvaluationEngine

An exemplary technique for managing the environmental conditions (e.g.,provide cooling for IT equipment) within a data center module isdescribed below with reference to data center module 300 of FIG. 3. Asdescribed above, one or more AHUs can be used to manage theenvironmental conditions (e.g., provide cooling for IT equipment) withina data center module. In various embodiments, a cold aisle temperatureset point (e.g., a desired air temperature in cold aisle 332) can beprovided. For example, a user 135 may use dashboard 535 to updatecentral database 510 with set point information. In various embodiments,DCICS 105 may derive or calculate a cold aisle temperature setpoint can,based on, for example, the cooling needs of IT equipment 305, thecomfort of operators of IT equipment 305, etc. AHU 310 can providechilled air 360 to maintain air 340 in cold aisle 332 at approximatelythe cold aisle temperature set point. In various embodiments, AHU 310can provide chilled air 360 to maintain air 340 in cold aisle 332 atapproximately the cold aisle temperature set point by controlling theamount of energy extracted (e.g., cooling) from heated air 345.

In various embodiments, aspects of the operation of AHU 310 can bemonitored and controlled by a DCICS (e.g. DCICS 105). For example, DCICS105 generate and communicate instructions to modulate a fan speed of AHU310 and a valve that controls the flow rate of chilled fluid (chilledfluid control valve or CCV) into AHU 310 and data center modulecontroller 400 can issue the instructions to the respective hardwaredevices. Modulating fan speed or CCV can affect the temperature of air340 in cold aisle 332. For example, modulating the CCV to increase theflow rate of chilled fluid into AHU 310 can lower the temperature of airleaving AHU 310 (e.g., chilled air 360), thereby lowering thetemperature of air 340 in cold aisle 332. As another example, modulatingthe fan speed of AHU 310 can increase the amount of air leaving AHU 310(e.g., chilled air 360), thereby lowering the temperature of air 340 incold aisle 332. In various embodiments, modulation of the fan speedand/or CCV can be used to maintain air 340 in cold aisle 332 atapproximately the cold aisle temperature set point.

FIG. 6 depicts a control system 600 for data center module 300. Asillustrated in FIG. 6, in various embodiments control system 600 is acascaded control loop. Proportional-integral-derivative controller (PID)605 can control CCV 610 (e.g., the CCV for AHU 310). For example, PID605 can receive a temperature set point for air leaving AHU 310 (e.g.,chilled air 360). PID 605 can receive the present temperature of airleaving AHU 310 (e.g., chilled air 360), e.g., from AHU outputtemperature and/or humidity sensor 355. PID 605 can modulate CCV 610 tomaintain the temperature of air leaving AHU 310 (e.g., chilled air 360)at approximately the temperature set point for air leaving AHU 310. PID615 can control fan motor 620 (e.g., the fan motor in AHU 310).

PID 625 can control PID 605 (e.g., by providing a set point temperaturefor air leaving AHU 310 (e.g., chilled air 360)). PID 630 can controlPID 615 (e.g., by providing a set point temperature for air leaving AHU310 (e.g., chilled air 360)).

Control system 600 can receive a set point 632 (e.g., cold aisletemperature set point). Control system 600 can receive a present value633 (e.g., the present temperature of air 340 in cold aisle 332 viatemperature and/or humidity sensor 350). Based on the difference betweenthe present value and the set point, PID 625 can increase or decreasethe set point provided to PID 605. For example, if the present valueexceeds the set point (e.g., the present temperature of air 340 in coldaisle 332 exceeds the cold aisle temperature set point), PID 625 canlower the set point temperature for air leaving AHU 310 provided to PID605. Based on the difference between the present value and the set point632, PID 630 can increase or decrease the set point provided to PID 615.

In various embodiments, lowering temperature of the chilled air 360 byincreasing the flow rate of chilled fluid into AHU 310 can be moreefficient and/or preferable to increasing the fan speed of AHU 310.Offset 635 can be used to increase the set point received by PID 630,thereby causing PID 630 to decrease the set point provided to PID 615 ata higher present value temperature than the present value temperature atwhich PID 625 can decrease the set point provided to PID 605.

Inhibit 640 can inhibit decreases in the temperature set point providedto PID 605 (e.g., temperature set point for air leaving AHU 310 (e.g.,chilled air 360)). In various embodiments, inhibit 640 can prevent thetemperature set point provided to PID 605 from going below a dew pointtemperature, a dew point temperature range, or a predetermined thresholdabove the dew point temperature. In various embodiments, inhibit 640 canprevent the temperature set point provided to PID 605 from going lowerwhen CCV is fully open. In various embodiments, when inhibit 640prevents decreases in the temperature set point provided to PID 605,inhibit 640 prevents PID 630 from increasing the temperature set pointprovided to PID 615.

In various embodiments, present value 633 can be the cold aisletemperature. In various embodiments, present value 633 can be the hotaisle temperature (e.g., the temperature of air 345 in hot aisle 335, asmeasured by temperature and/or humidity sensor 352). In variousembodiments, present value 633 can be the planar or processortemperature of IT equipment. In various embodiments, switch 650 cancontrol whether present value 633 is from an external source (e.g.,sensor measuring the temperature of IT equipment planar or processor) orinternal source (e.g., sensor measuring the cold aisle temperature).

While the described embodiment controls AHU 310, it should beappreciated that control system 600 can control the operation of one ormore data center assets (e.g., IT infrastructure hardware). In variousembodiments, a module (e.g., module 225 of FIG. 1) can contain multipleAHUs (e.g., 4, 6, or 8 AHUs). Control module 600 can control the CCV andfan motor for each of the AHUs as described above. In variousembodiments, a module can be divided into zones (e.g., divided into twozones) that can each operate as described above (e.g., the operation ofAHUs in each zone are controlled separately from AHUs in other zonesbased on set points for each zone).

In various embodiments, an AHU can be “In Service” or “Out of Service.”An “In Service” AHU can have its fan speed set to greater than 0% (e.g.,the fan is rotating) and the CCV can be set to greater 0% (e.g., thevalve permits chilled fluid to enter the AHU). An “Out of Service” AHUcan have its fan speed set to 0% (e.g., the fan is not rotating) and theCCV can be set to 0% (e.g., the valve is closed). In various embodimentsone or more AHUs can be “In Service” and operate as described above. Invarious embodiments, when inhibit 640 prevents decreases in thetemperature set point provided to PID 605, one or more “Out of Service”AHUs can be changed to “In Service,” thereby increasing the coolingcapacity available.

In various embodiments, control system 600 can be housed in a datacenter module (e.g., in pressurized plenum 330 of data center module300). In various embodiments, a data center module controller (e.g.,data center module controller 400) can implement control system 600. Invarious embodiments, control system 600 can be a part of DCICS 105.

FIG. 7 is a flow chart 700 depicting a method of operating a data centermodule. At step 705, an air pressure differential is measured. Forexample, with reference to data center module 300, pressure sensors 360can measure the differential pressure between cold aisle 332 and hotaisle 335. At step 710, it is determined if the pressure differential iswithin a predetermined range. For example, a particular pressuredifferential can facilitate air flowing from cold aisle 332, though ITEquipment 305. If the pressure differential is not within thepredetermined range, the fan speed in one or more AHUs can be adjustedat step 715. For example, if the measured pressure differential betweencold aisle 332 and hot aisle 335 is below the predetermined range, thespeed of the fan in AHU 310 can be increased, thereby increasing thepressure in cold aisle 332. If the measured pressure differentialbetween cold aisle 332 and hot aisle 335 is above the predeterminedrange, the speed of the fan in AHU 310 can be decreased, therebydecreasing the pressure in cold aisle 332.

If the pressure differential is within the predetermined range, themethod proceeds to step 720 where the temperature is measured. Forexample, the temperature of air 340 in cold aisle 332 can be determinedvia temperature and/or humidity sensor 350. At step 725, it isdetermined if the temperature is within a predetermined range. Forexample, a desired air temperature range in cold aisle 332 can beidentified (e.g., a cold aisle temperature set point). If the measuredtemperature is not within the predetermined range, the CCV can beadjusted at step 730. For example, as described above, modulating theCCV to increase the flow rate of chilled fluid into AHU 310 can lowerthe temperature of air leaving AHU 310 (e.g., chilled air 360), therebylowering the temperature of air 340 in cold aisle 332. If the measuredtemperature is within the predetermined range, the method returns tostep 705.

The technology described herein can include techniques for monitoring,analyzing and controlling a data center. As described above, DCICS 105can access data center operations data associated with, for example datacenter infrastructure 110, IT hardware 115, virtualization layer 120,operating system 125, application 130, and user 135 shown in FIG. 1. Thedata center operations data can be analyzed, thereby permittingreal-time or near real-time monitoring of the operation of the datacenter.

In various embodiments, DCICS 105 accesses data associated with variousoperational systems, or subsystems, of a data center infrastructure.Such systems include environmental, information technology, powerdelivery, energy recovery, etc. Data is accessed via “data collectionpoints” which, in various embodiments, may include data sources,sensors, detectors, status indicators, operational control indicators,mechanical settings, IT hardware characteristics and state information,software system attribute and setting parameters, etc. For example, suchdata access points can include: controllers, humidity sensors, firedetectors, smoke detectors, high sensitivity smoke detectors, very earlysmoke detection apparatuses, moisture detectors, spot leak detectors,pressure sensors, intrusion detectors, valve indicators (e.g., controlreturn valve position indicators, supply valve position indicator,etc.), power input monitors, power output monitors, fan speed indicatorsand fan direction indicators.

In various embodiments, such environmental systems include: FireDetection (e.g., alarm, high sensitivity smoke detector (“HSSD”)); FireSuppression: (alarm, panel reset); Temperature (e.g., internal,external, cold aisle, hot aisle, etc.); Pressure (e.g., moduledifferential pressure); Humidity (relative humidity, dew point, controlhumidity); and Lead detection (spot leak detectors).

In various embodiments, such IT systems include: Server (e.g., serverCPU, RAM, disk, fan speed, temperature); Network (e.g., throughput,routing information, firewalls, alerts, SNMP based); Virtualization(e.g., manage and monitor both virtual workloads); IT hardwarecharacteristics and state information; and Software system attribute andsetting parameters.

In various embodiments, such power delivery systems include: Generators(e.g., generator load and status, fuel level, oil pressure, remotecontrol stop and start, etc.); UPS (e.g., input and output, total loadof UPS, power factor; PDU (e.g., branch circuit monitoring, input/outputload, etc.); and Switchgear (e.g., input/output bus, demand current,power factor, etc.).

In various embodiments, such energy recovery systems include: Chillers(e.g., supply and return temperature, system pressure, control outlettemperature, set point, etc.); Air Handlers (e.g., inlet/outlettemperature, control coil demand, fan mode, fan speed, fan on/off,filter hours, etc.); and Valves (e.g., heat removal, differentialpressure, flow, valve position, control return valve position, supplyvalve position, etc.).

The technology can include techniques for specifying how data centeroperations data is identified, aggregated and analyzed. In variousembodiments, the technology includes a scripting language for specifyinghow data center operations data is aggregated and analyzed along withaggregating and analyzing the data as specified. In various embodiments,an expression in the language for specifying how data center operationsdata is aggregated and analyzed can have the following syntax:

-   -   [input set]|>operation

In the expression above, input set is the source (e.g., set of datacollection points in the data center) from which to collect data centeroperations data to aggregate and analyze, and operation is the operationto perform on the input set. input set can be specified by, for example,data collection point type, device type, partition membership,containing system, etc. from which to collect the data center operationsdata. The input set can include other expressions. Example input setsfollow. In the examples, the input set members are specified usingUniform Resource Names, but in various embodiments other manners ofidentifying the input set members can be used.

[urn:NID:sensortype:Temp]—specifies the input set as data from alltemperature sensors in the data center.

[urn:NID:partition:1918 & urn:NID:sensortype:Temp]—specifies the inputset as data from all temperature sensors in partition 1918.

[urn:NID:device:ba3e38b9-de9d-e011-ba7c-005056876935 &(urn:NID:sensortype:136|urn:NID:sensortype:1086)]—specifies the inputset as data from all power meters of type 135 and 1086 in the PDUidentified as urn:NID:device:ba3e38b9-de9d-e011-ba7c-005056876935.

In the expression above, operation is the operation to perform on thedata from the input set. For example, the operation can be amathematical equation, algorithm or stochastic method. In variousembodiments, operations include SUM (e.g., the summation of all datafrom sensors in input set), MAX (e.g., the maximum value of all datafrom sensors in input set), MIN (e.g., the minimum value of all datafrom sensors in input set), MEAN (e.g., the mean of all data fromsensors in input set), STDEV (e.g., standard deviation) and COUNT (e.g.,the number of data items from sensors in input set). Performance of theoperation on data from the data collection points in input set canproduce a result (e.g., a numeric value).

In various embodiments, the result of an expression of the form [inputset]|>operation can be used in an arithmetic operation, such asaddition, subtraction, division, or multiplication. In variousembodiments, data from a single data collection point (e.g. a singlesensor) or a plurality of data collection points can be used in anarithmetic operation with the result of an expression of the form [inputset]|>operation.

DCICS 105 can include an execution engine for performing the aggregationand analysis specified by an expression. The technology can receive anexpression specifying how to aggregate and analyze data centeroperations data (e.g., in the syntax above) and aggregate and analyzethe data as specified. In various embodiments, DCICS 105 can receive anexpression (e.g., as a text string, in any non-compiled form, etc.). Theexpression can indicate a set of data collection points within one ormore of data center 550, 560, and/or 570 and an operation to perform ondata from those sensors. In various embodiments, DCICS 105 can generatemachine code instructions based on the expression (e.g., by just-in-timecompilation), wherein the machine code instructions include instructionsto aggregate and analyze the data as specified.

DCICS 105 can execute the machine code instructions, generating a resultof the operation. In various embodiments, DCICS 105 can query a databasecontaining data center operations data (e.g., central database 510) tocollect data from data collection points specified in the input set. Invarious embodiments, the DCICS 105 can collect the data directly fromthe data collection points (e.g. directly from a sensor, from a sensorvia a translator, from a sensor via an OPC client, etc) specified in theinput set.

In various embodiments, DCICS 105 can approximate data and populate gapsin data time series. In some operational configurations, data from datacollections points is collected and/or populated in databases (e.g.Central Database 510) upon the occurrence of an event. For example, datafrom a temperature sensor may be collected in response to a temperaturereading at the sensor varying from a previous reading by some threshold,percentage and/or statistical variance. In this example, data for aparticular data collection point may be populated on an infrequentbasis. However, DCICS 105, its associated modules, associated databases,interfacing applications, interfacing network elements or networkhardware may, in some embodiments, be configured to expect data to bepopulated on a predictable or deterministic schedule. For example, aninterfacing application may want to correlate the value at time (t) ofthousands of different data points and/or an interfacing system may beconfigured to “expect” data to be populated for each and every of the1440 minutes in a day.

However, as described above, in some embodiments, data may not have beenpopulated for certain data collection points for every possible time(t). DCICS 105 is configured with logic and methods to predict andpopulate data in response to data collection point data having “gaps.”In various embodiments, DCICS 105 may access prior and subsequent datafor the data collection point(s) being analyzed in order to help predicta value that may have occurred in an interim or “gap” period. In variousembodiments, DCICS 105 may access related or associated data (e.g.,sensors that are determined to have been in analogous operationalconditions during a time period) to help predict a value that may haveoccurred in an interim or “gap” period. In various embodiments, DCICS105 may use statistical tools (e.g., linear and non-linear regression),distribution modeling, forecasting and/or simulation methods todetermine data values. In various embodiments, DCICS 105 may utilizedatabase management system capabilities or tools to determine and/orpopulate data; for example: DCICS 105 may make use of a database triggerto initiate updates to data based upon a schedule and/or the occurrenceof defined events, DCICS 105 may make use of a Cartesian join topopulate gaps in rows of data between two different data points.

In addition to identifying, analyzing, predicting and populating datawhen there are gaps in data, in various embodiments, DCICS 105 caneliminate, aggregate, summarize and/or prioritize data. For example, invarious operational scenarios, data for data collection points may begenerated on a frequent schedule and/or in response to events that occurfrequently. For example the utilization of a CPU running in a datacenter may be measured every tenth of a second. However, DCICS 105, itsassociated modules, associated databases, interfacing applications,interfacing network elements or network hardware may, in someembodiments, be configured to expect such data to be populated on apredictable or deterministic schedule that is less frequent than therate or density with which the data is populated. In variousembodiments, DCICS 105 is configured with logic and methods toaggregate, summarize and/or prioritize data in response to datacollection point data being too frequently populated and/or denselyoccurring. In various embodiments, DCICS 105 may access current, priorand subsequent data for the data collection point being analyzed inorder to help determine a value to be used for a particular time (t) ora particular set of times. In various embodiments, DCICS 105 may accessrelated or associated data (e.g., sensors that are determined to havebeen in analogous operational conditions during a time period) in orderto help determine a value to be used for a particular time (t) or aparticular set of times. In various embodiments, DCICS 105 may usestatistical tools (e.g., linear and non-linear regression), distributionmodeling, forecasting and/or simulation methods to determine datavalues. In various embodiments, DCICS 105 may utilize databasemanangement system capabilities or tools to determine and/or populatedata; for example: DCICS 105 may make use of a database trigger toinitiate updates to data based upon a schedule and/or the occurrence ofdefined events, DCICS 105 may make use of a Cartesian join to populategaps in rows of data between two different data points.

In various embodiments, the result can be interpreted by DCICS 105,shown on a display and/or stored to a data base (e.g., central database510). In various embodiments, DCICS 105 determines a control instructionbased upon the result and executing the control instruction causes achange in the operational state of a data center and/or a data centerasset (e.g. data center system, IT hardware, software, etc.).

In various embodiments, DCICS 105 can update the result of an operationwhen underlying data changes. For example, if the expression specifiesaveraging all temperature data from all temperature sensors in a datacenter module, DCICS 105 can update the result of the operation wheneverdata from one or more of the temperature sensors changes. In variousembodiments, DCICS 105 determines that point data from at least one ofthe data collection points has changed and executes, based upon thepoint data, machine code instructions to produce a second result of anoperation. DCICS 105 may further determine, based upon the secondresult, a second control instruction.

Embodiments of this disclosure include systems and methods ofmonitoring, analyzing and controlling data centers and related hardwareand software. In an embodiment, a DCICS determines, based upon a firstexpression, a first operation and data collection points associated withdata center assets. Data collection points may include sensors,indicators, detectors, system data, etc. In various embodiments, theexpression may be obtained from a database, received via an API,received via user input, received or obtained from another system or ahardware device, etc. In various embodiments, the expression may bederived or generated by the DCICS based upon, for example, otherexpressions, data or events.

DCICS interprets and/or analyzes the expression and generates machinecode instructions that may include, for example, instructions to obtainthe first point data from the data collection points and to perform theoperation based upon the first point data. In various embodiments, theinstructions may include instructions for obtaining, receiving orretrieving the first point data. In some situations, the operation maybe recursive and/or dependent upon the first point data. For instance,the operation itself may iterate and one or more interactions maythemselves be dependent upon a nested calculation, updated data,simulated data, a previous iteration, etc.

DCICS 105 executes the machine code instructions to produce a firstresult of the first operation. In various embodiments, in response toproducing the first result, DCICS 105 examines or analyzes the firstresult and determines a first control instruction. Executing the firstcontrol instruction causes a change in an operational state of at leastone of a data center asset and a data center.

In various embodiments, determining the first control instruction, maycomprise generating the first control instruction, analyzing historicaldata, analyzing external data, performing a forecast, simulation orother predictive analytics, performing an optimization, receiving userinput, etc. For example, DCICS 105 may process an expression (or aplurality of expressions) that is associated with the processingcapability (e.g. load, utilization, etc.) on a particular set of datacenter assets such as servers, storage devices and/or networkingdevices. DCICS 105 analyzes the result of the operation associated withthe expression and may determine that those data center assets are atrisk of failing or performing below a required performance threshold.DCICS 105 may examine, for example, one or more of historical data, datacenter asset profile information, performance data, capability data,service level agreement data and performance needs forecasts.

Based upon this analysis, DCICS 105 may determine that a new operationalstate of the data center that is achieved by moving a softwareapplication to a different set of data center assets would help mitigateoperational risk. DCICS 105 may identify a new or preferredconfiguration, identify a workflow to implement the change and generateinstructions for executing such a change.

In various embodiments, DCICS may at least one of execute, partiallyexecute the control instruction, and communicate the control instructionto a hardware device (e.g. controller 400) and/or software program forexecution.

In various embodiments, the control instruction may include machine codeinstructions, an API call, an electrical signal, a trigger, object code,script, etc. Executing the control instruction causes a change in theoperational state of a data center, data center asset and/or or a datacenter system. The control instruction may include instructions thatcause, for example, moving data from a first data storage device to asecond data storage device, moving a software application from a firstcomputing device to a second computing device, terminating a softwareapplication, initiating a software application, initiating a virusprotection strategy, implementing an intrusion prevention strategy,creating a user permission, revoking a user permission, shutting down acomputing device, adjusting an operating temperature, initiating a firesuppression system, turning on an air handler, shutting down an airhandler, adjusting a fan speed, adjusting a valve, locking a door,unlocking a door, locking an access panel, and unlocking an accesspanel.

In various embodiments, moving a software application from a firstcomputing device to a second computing device may include the firstcomputing device in a separate physical location from the firstcomputing device, the first computing device and/or the second computingdevice being virtual machines, and/or creating a virtual computingdevice.

DCICS 105 can verify the change in the operational state. For example,DCICS 105 obtains data from data collection points, analyzes the dataand determines whether the desired operational state has been achieved.In various embodiments, DCICS 105 may generate an expression forverifying that the operational state has been changed. Generating anexpression may include one or more of deriving, calculating, orreceiving the expression.

In various embodiments, verifying the change in the operational statemay include determining a control operation result and/or analyzingsecond point data, where the control operation result is based upon theexecuting the first control instruction and where the second point datais obtained from the data collection points. The control operationresult might be determined and/or obtained, for example, from thecontroller, a database and/or data collection points. In variousembodiments, DCICS 105 may analyze the control operation result, thesecond point data and/or the change in the operational state, anddetermine a second operation or a second expression.

In various embodiments, DCICS 105 can use, for example, one or more of abusiness rule, a proprietary algorithm, external input, an interfacingsystem, artificial intelligence, simulations and other deterministic orstochastic predictive analytics to determine operational states of datacenters or data center assets, evaluate or predict data center assetattributes (e.g., risk state, power capability, physical space, coolingcapability, networking capability, redundancy capability, geographicproximity, etc.).

DCICS 105 can develop alternative strategies for transitioning to anoptimal (or preferred) operational state and/or minimizing risk for datacenter assets such as, for example, a data center, a plurality of datacenters, a partition, an application, a functional capability (e.g.combination of application and performance capabilities), anenterprise's IT infrastructure, etc. Such data center assets may bephysically located in multiple data centers and/or in geographicallydiverse locations. In developing such alternative strategies, DCICS 105can access profile, product, configuration, sales, utilization,performance, constraint and data center asset attribute data from adatabase such as central database 510, BI Warehouse 515, CRM 505,accounting 507, etc. In various embodiments, DCICS 105 includes acomplete set of data entry, design and maintenance graphical userinterfaces that enable design, entry, maintenance and visualization ofdata center asset information.

Based upon, for example, data collected, determined operational statesand alternative strategies developed, DCICS 105 can direct controlmechanisms to manipulate operational conditions and configurations andtransition data center assets (e.g., hardware and software) to newoperational states. For example, DCICS 105 may send an instruction todata center module controller 400, to translator 107, to OPC client 537and/or directly to a data center hardware or software component.

In general, DCICS 105 can collect data from one or more data sources,where each data source of the one or more data sources relates to theoperation of the data center. For example, as described above, the DCICS105 can collect, access and/or obtain data from data centerinfrastructure 110, IT hardware 115, virtualization layer 120, operatingsystem 125, application 130, and user 135. Based on the collected data,the technology can identify a current operational state of the datacenter. Generally, an operational state refers to a past, present orfuture status of a data center asset or component, a partition orgrouping of data center assets, an entire data center, and/or multipledata centers.

In various embodiments, DCICS 105 determines one or more alternativeoperational states for the data center based on, for example, thecurrent operational state, a portion of the collected data, and one ormore pre-defined criteria. DCICS 105 can identify a desired operationalstate of the one or more alternative operational states. The technologycan utilize one or more data center controls to transition the datacenter to the desired operational state.

DCICS 105 employs intelligent control to optimize data centeroperations. In various embodiments, DCICS 105 accesses inventory data,operational data, historical data and external data (collectively,“input data”), determines data center optimization dimensions, producespredictive data, determines an optimization action based upon datacenter optimization dimensions, the predictive data and input data,generates an optimization instruction based upon the optimizationaction. Execution of the optimization instruction causes a change in anoperational state associated with the data center operations. In variousembodiments, DCICS 105 is configured to collect data in order to verifyand/or validate the change in the operational state.

Data center optimization dimensions may include, for example,reliability, redundancy, resiliency, risk, latency, efficiency, cost andthreat avoidance. In various embodiments, a redundancy rating of 2N,N+1, N, etc. may be assigned to certain power and cooling capabilities.

In various embodiments, accessing input data comprises identifying andanalyzing the input data. For example, identifying input data mayinclude receiving input, obtaining from a data source, calculating andforecasting. The data source may include, for example, a historicaldatabase (e.g., central DB 510, BI Warehouse 515), a customerrelationship management database (e.g., CRM 505), a data warehouse(e.g., BI Warehouse 515), a product catalog, an accounting system, theinternet, a government data source, a vendor database, a utility system,a power system, a security system, a network provider system, and datacollection points associated with the data center operations.

Analyzing the input data may include analyzing a plurality of datacenter assets based upon optimization dimensions. For example, DCICS 105may determine that a suite of applications enabled by a data centershould be moved in order to reduce latency, balance a load and free upsome physical space. DCICS 105 may analyze a number of data centerassets based upon optimization dimensions (e.g., reliability,resiliency, risk, latency, efficiency, cost and threat avoidance) andbased upon data center asset attributes (e.g., risk state, powercapability, physical space, cooling capability, networking capability,redundancy capability, and geographic proximity).

In various embodiments, analyzing the input data includes retrieving,mining, parsing, validating, obtaining and/or analyzing the input datato identify at least one of customer data, configuration data, productdata, sales data, service level agreement (SLA) data, processingrequirements, storage requirements, network bandwidth requirements, costdata, revenue data, and revenue by product data.

In various embodiments, DCICS 105 can identify a plurality ofoptimization actions. Identifying an optimization action can includeDCICS 105: performing a proprietary algorithm; performing a simulation;performing artificial intelligence; performing a stochastic method;receiving user input; and executing a linear programming model. Basedupon the optimization dimensions, DCICS 105 may determine anoptimization score for each of the identified optimization actions. Forexample: a redundancy optimization dimension analysis may ensure thatalternative data center assets under consideration have sufficientcooling and/or power redundancy ratings in order to support processing,storage and network capacity needed; a latency optimization dimensionanalysis may ensure that alternative data center assets underconsideration have sufficiently high network bandwidth capability andlow latency estimations in order to support business objective and/orservice level agreements.

In various embodiments, DCICS 105 monitors and controls physicalsecurity of data centers. For example, the DCICS 105 can monitor thephysical security of a partition, analyze collected data, performforecasts, simulations, and risk analysis, develop scenarios forimproving the operational state, pick a preferred scenario, develop aset of actions (e.g. workflow) to implement the new operational state,and execute or communicate control instructions to implement the actionsand adjust the operational state (e.g., to improve physical security).

Individuals and/or groups of individuals can be assigned physical accessrights to data centers and data center assets. Exemplary rights includeentry through exterior doors of the data center, entry through interiordoors of the data center, entry through data center module doors, accessto IT Hardware cabinets, access to particular infrastructure equipment,etc.

DCICS 105 can obtain data center operations data associated with thephysical security of a partition and a current operational state. Forexample, DCICS 105 can collect data about physical barriers (such aswhich doors are opened/closed or locked/unlocked), IT hardware cabinets(such as which cabinets or panels are opened/closed or locked/unlocked),and the presence of individuals in the partition (e.g., by use of facialrecognition, biometrics, card reader, etc.). DCICS 105 can also identifyaccess rights data for recognized individuals. DCICS 105 can determinethat the person attempting to access a partition has been assignedaccess rights to the partition and should be allowed physical access.DCICS 105 may alter the operational state of the partition by unlockinga door and/or cabinet. In various embodiments, DCICS 105 can determinethat the person attempting to access the partition has not been assignedaccess rights to the partition and should not be allowed physicalaccess.

Based on the data center operations data, DCICS 105 can cause one ormore changes, such as: grant/restrict access to perimeter gates,entrance doors, gates, service corridors, service galleries, rooms,cages; lock doors, service hatches, racks housing IT hardware; lockvarious service equipment including power distribution units,controllers, air handling units; trigger alerts to administrators; andlog equipment accessed and activity inside the partition.

In various embodiments, the technology can monitor for a naturaldisaster (flood, earthquake, fire, etc.) and adjust the operation thedata center based on the data collected. In various embodiments, DCICS105 can monitor for natural disasters that can affect the operation ofthe data center. For example, using sniffers connected to smoke and/orfire detection sensors, DCICS 105 can detect fires and formulateappropriate response actions, identify an optimal action and utilizecontrol functions to implement the action. In various embodiments, DCICS105 can detect earthquakes from vibration detectors in the monitoredpartition, from an external information source, such as U.S. GeologicalService (USGS) seismic data, and/or by performing a risk analysis orforecast based detected data, user input, external information, forecastmodules and historical data. DCICS 105 can detect moisture or floodsusing leak sensors in a partition or from an external informationsource, such as external weather data.

In various embodiments, if a natural disaster is predicted or detected,DCICS 105 can determine the type or category of the disaster (e.g.,fire, flood, earthquake, etc.) and determine alternative data centerassets (e.g., computing, storage, networking, etc. assets) available toprovide services (memory, CPU, storage, network connectivity) toapplications running in the partition affected by the natural disaster.

In various embodiments, DCICS 105 can rank the alternative data centerassets based on one or more of user, device, application, IT hardware,business rules (e.g. a priority assignment) and/or data center rules anyof which may be retrieved from a database, received from a system,derived as part of an analysis, determined based upon a pre-determinedassignment, received via user input, etc.

DCICS 105 can present the alternative data center assets to the operatorof the partition affected by the natural disaster to facilitatedynamically transferring the applications running in the partitionaffected by the natural disaster to one or more of the alternative datacenter assets. In various embodiments, DCICS 105 can cause such atransfer without operator interaction (e.g., systematic, automatedtransfer to the identified alternative data center asset(s)).

In various embodiments, DCICS 105 can determine potential alternativedata center assets based on and/or by determining one or more of thefollowing: risk level (e.g., determine whether the alternative datacenter asset is not also under an unacceptable risk state (fire, flood,etc.)); whether the alternative data center asset has sufficient powerto support the processing/storage/networking, etc. needed; whether thealternative data center asset has sufficient storage capacity to supportthe processing/storage/networking needed; whether the alternative datacenter asset has sufficient cooling available to support theprocessing/storage/networking needed; whether the alternative datacenter asset has sufficient networking to support theprocessing/storage/networking needed;

whether the alternative data center asset has sufficient power and/orcooling redundancy (e.g., 2N, N+1, N, etc.) to support theprocessing/storage/networking needed; and whether the alternative datacenter asset is within an acceptable geographical proximity to thepartition affected by the natural disaster. In various embodiments, anyof the above can be a binary value (e.g., yes or no). In variousembodiments, any of the above can be a range of values.

The technology can monitor logical security (e.g., monitor for networkintrusion). In various embodiments, DCICS 105 receives data centeroperations data relating to intrusions from an intrusion detectionsystem (IDS). The IDS can, for example, use pattern matching on routerlogs to detect logical intrusions (or logical intrusion attempts). DCICS105 can determine alternative data center assets (e.g., computing,storage, networking, etc. assets) available (e.g., with current and/orfuture capacity) to accommodate applications running in the partitionaffected by the intrusion.

In various embodiments, DCICS 105 can rank the alternative data centerassets based on a priority assignment determined by the operator of thepartition affected by the logical intrusion. In various embodiments,DCICS 105 can rank the alternative data center assets based on one ormore of user, device, application, IT Hardware, business rules and/ordata center rules. In various embodiments, DCICS 105 can present thealternative data center assets to the operator of the partition affectedby the logical intrusion to facilitate dynamically transferring theapplications running in the partition affected by the logical intrusionto one or more of the alternative data center assets. In variousembodiments, DCICS 105 can cause the transfer without operatorinteraction (e.g., by automatically transferring to the highest rankedalternative data center assets).

In various embodiments, DCICS 105 can determine potential alternativedata center assets based on one or more of the following: assetutilization, utilization forecasts, physical security, logical security,current latency; utility costs; power capacity or availability, coolingcapability, physical space, network providers, network bandwidth,network redundancy, power redundancy, a scope or type of intrusion(inbound, outbound, application based, networked based, etc.), knowntechniques used to terminate the intrusion, present intrusion evasiontechniques including alternative data center assets which may have alower risk profile.

In various embodiments, DCICS 105 can mitigate the risk of intrusion by,for example, shutting down network connectivity, issuing alarms topartitions affected by the logical intrusion; alerting users ofpartitions affected by the logical intrusion; etc.

The technology can monitor latency (e.g., the network latencyencountered by applications running on IT equipment in the data center).In various embodiments, DCICS 105 can receive data center operationsdata relating to latency from network monitors. DCICS 105 can determinealternative data center assets (e.g., computing, storage, networking,etc. assets) available (e.g., with capacity) to accommodate applicationsrunning in the partition affected by high latency. In variousembodiments, DCICS 105 can rank the alternative data center assets basedon a priority assignment determined by the operator of the partitionaffected by the high latency. In various embodiments, DCICS 105 can rankthe alternative data center assets based on one or more of user, device,application, IT Hardware, business rules and/or data center rules. DCICS105 can present the alternative data center assets to the operator ofthe partition affected by the high latency to facilitate dynamicallytransferring the applications running in the partition affected by thehigh latency to one or more of the alternative data center assets. Invarious embodiments, DCICS 105 can cause the transfer without operatorinteraction (e.g., by automatically transferring to the highest rankedalternative data center assets).

In various embodiments, DCICS 105 can determine potential alternativedata center assets based on one or more of the following: assetutilization, utilization forecasts, physical security, logical security,current latency; utility costs; power capacity or availability, powerutilization effectiveness, cooling capability, physical space, networkproviders, network bandwidth, network redundancy and power redundancy.

In determining alternative data center actions in order to optimize alatency optimization dimension, DCICS 105 may obtain information fromany layer of the IT stack. The IT stack may include, for example, a userlayer, a virtualization layer, an application layer, an operating systemlayer, an IT or hardware layer and a data center infrastructure layer.

In various embodiments, the technology can monitor redundancy (e.g., theredundancy of AHUs within a data center module). In various embodiments,a data center management system can maintain a level of AHU redundancy(e.g., N, N+1, 2N, 2N+1) in a data center module by selectively enablingAHUs based on current power utilization in the data center module. Forexample, if each AHU can provide cooling for 35 kW of IT equipment loadand the data center module is maintaining 2N AHU redundancy, the datacenter module can enable 2 AHUs under 50 kW of IT equipment load and 3AHUs under 80 kW of IT equipment load.

The technology can involve acoustic monitoring. In various embodiments,DCICS 105 can receive acoustic data from acoustic sensors in the datacenter. In various embodiments, DCICS 105 can compare known “good”acoustic profiles to the received acoustic data, and identify datacenter equipment as possibly failing or failed if its acoustic profilediffers from the known “good” acoustic profiles. In various embodiments,DCICS 105 can compare known “bad” acoustic profiles to the receivedacoustic data, and identify data center equipment as possibly failing orfailed if its acoustic profile matches the known “bad” acousticprofiles.

In various embodiments, DCICS 105 includes a manufacturing resourceplanning (MRP) tool for identifying, extracting and analyzing datacenter operational, business and financial data in order to plan futuredata center operations. Thus, DCICS 105 provides data center plannersand operators a real-time tool which enables them to increase capacityand/or capability in a just-in-time manner.

DCICS 105's MRP tool may generate data useful in such planning efforts.For instance: DCICS 105 may set flags indicating when to add coolingcapacity at an individual data center asset (e.g. data module) level. Invarious embodiments, such a cooling capacity flag may be set based upon,for example, an air handling unit capacity reaches some capacity of arunning average of the overall power consumption within the data center.As disclosed previously, an expression may be used to define datacollection and operations to support such an analysis. In variousembodiments. DCICS 105 may track available power at various points in adata center. Such tracking may include granular data collection such ascollecting data regarding power pulled from a substation to individualbranch circuits within a data center power distribution infrastructure.Such data will allow DCICS 105 to produce recommendations for increasinga data center infrastructure capacity; e.g. order more data modules,power modules, UPSs, PDUs, etc.

The technology can monitor efficiency (e.g., the power usageeffectiveness for a partition). In various embodiments, DCICS 105 canreceive data center operations data relating to power consumption. DCICS105 can determine alternative data center assets (e.g., computing,storage, networking, etc. assets) available (e.g., with capacity andappropriate capability) to takeover any applications running in thepartition and with lower power usage effectiveness (e.g., as calculatedbelow).

In various embodiments, a power usage effectiveness (PUE) can becalculated. PUE can be a measure of how efficiently a data centeroperates. PUE can be defined as the ratio of total power consumed (e.g.,total power consumed by a data center) to the total power made availableto specified IT equipment for productive use (e.g., the total power madeavailable to servers for operating and running applications). Thetechnology facilitates determining PUE for a data center as a wholeand/or for partitions within the data center. For example, PUE can bedetermined for one or more data modules, one or more racks of serverswithin one or more data modules, or an individual server within a datamodule. More generally, PUE can be determined for any partition withinthe data center.

As described above, DCICS 105 can collect data center operations data,including data related to power generation and power consumption by thevarious components of the data centers that it monitors. FIG. 8 depictsmodular data center 800. Modular data center 800 includes chiller module810, power module 820, and data module 830. Data module 830 can containIT equipment 840. When data center 800 is in operation, chiller module810 can provide chilled fluid to power module 820 and data module 830,facilitating cooling of the equipment contained therein (e.g., AHUswithin power module 820 and data module 830 can utilize the chilledfluid to provide cooling as described above). Power module 820 canreceive power from a power source (e.g., electric utility, generator,etc.). Power module 820 can distribute power to data module 830. Datamodule 830 can provide power to IT equipment 840 for productive use.

In various embodiments, DCICS 105 can monitor chiller module input 845(e.g., kW received by chiller module 810 from an electric utility) andchiller module output 850 (e.g., cooling capacity of the chilled fluidprovided by chiller module 810). DCICS 105 can monitor power moduleinput 855 (e.g., kW received by power module 820 from an electricutility). DCICS 105 can monitor power module output 860 (e.g., kWreceived by data module 830 from power module 820). DCICS 105 canmonitor Delivered IT Power 865 (e.g., kW delivered to the IT equipment840 for productive use).

DCICS 105 can calculate PUE for any data center asset that receivespower for productive use (e.g., IT Equipment 840), based on theDelivered IT Power to that IT equipment. In various embodiments, Eq. 1can be used to calculate PUE for the specified IT equipment:

$\begin{matrix}{{PUE} = \frac{\begin{matrix}{{\left( {{PMB}*{DMB}*{CMB}} \right)*{DITP}} +} \\{\begin{pmatrix}{\left( {\left( {{PMB}*{DMB}} \right) - 1} \right)*} \\{DITP}\end{pmatrix} + {DITP}}\end{matrix}}{DITP}} & {{Eq}.\mspace{14mu} 1}\end{matrix}$

Where:

CMB is the Chiller Module Burden, provided by the equation:

Chiller Module Input (kw)/Chiller Module Output (kw)

DMB is the Data Module Burden, provided by the equation:

Data Module Input (kw)/Data Module Output (kw)

PMB is the Power Module Burden, provided by the equation:

Power Module Input (kw)/Power Module Output (kw)

DITP is the Delivered IT Power (e.g., the power delivered to thespecified IT equipment that receives the DITP for productive use).

Referring to the example illustrated in FIG. 8,

CMB=chiller module input 845/chiller module output 850=20 kW/100 kW=0.2

PMB=power module input 855/power module output 860=1.05

DMB=power module output 860(e.g., data module input)/Delivered IT Power865 (e.g., data module output)=1.02

DITP=Delivered IT Power 865=93 kW

Evaluating Eq. 1 with the values in the example above results in a PUEcalculation of 1.29 (rounded to the nearest tenth). It should beappreciated that PUE can be calculated for any IT equipment (e.g.,partitions including one or more servers) based on the Delivered ITPower provided to the IT equipment.

The above-described techniques can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The implementation can be as a computer programproduct, i.e., a computer program tangibly embodied in an informationcarrier, e.g., in a machine-readable storage device, for execution by,or to control the operation of, data processing apparatus, e.g., aprogrammable processor, a computer, or multiple computers. A computerprogram can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand-alone program, application or engine, or as amodule, component, subroutine, or other unit suitable for use in acomputing environment. A computer program can be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a communication network.Computer programs are configured to enable online and automatedfunctions such as, for example, sending and receiving messages,receiving query requests, configuring responses, dynamically configuringuser interfaces, requesting data, sending control instructions,receiving data, parsing data, displaying data, executing complexprocesses, interpreting scripts, constructing database queries,executing data base queries, executing simulations, calculations,forecasts, mathematical techniques, workflows and/or algorithms,prompting users, verifying user responses, initiating processes,initiating other computer programs, triggering downstream systems andprocesses, encrypting and decrypting.

Computer programs and other software elements may be loaded onto ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions that execute on the computer or other programmable dataprocessing means for implementing the functions specified in theflowchart block or blocks. These computer program instructions may alsobe stored in a computer-readable memory that can direct a computer orother programmable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the function specified herein or in flowchart block orblocks. The computer program instructions may also be loaded onto acomputer or other programmable data processing apparatus to cause aseries of operational steps to be performed on the computer or otherprogrammable apparatus to produce a computer-implemented process suchthat the instructions which execute on the computer or otherprogrammable apparatus provide steps for implementing the functionsspecified in the flowchart block or blocks.

Functional blocks of the block diagrams and flowchart illustrationssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions, andprogram instruction means for performing the specified functions. Itwill also be understood that each functional block of the block diagramsand flowchart illustrations, and combinations of functional blocks inthe block diagrams and flowchart illustrations, can be implemented byeither special purpose hardware-based computer systems which perform thespecified functions or steps, or suitable combinations of specialpurpose hardware and computer instructions. Further, illustrations ofthe process flows and the descriptions thereof may make reference touser windows, web pages, web sites, web forms, prompts, etc.Practitioners will appreciate that the illustrated steps describedherein may comprise in any number of configurations including the use ofwindows, web pages, web forms, popup windows, prompts and/or the like.It should be further appreciated that the multiple steps as illustratedand described may be combined into single web pages and/or windows buthave been expanded for the sake of simplicity. In other cases, stepsillustrated and described as single process steps may be separated intomultiple web pages and/or windows but have been combined for simplicity.

Method steps can be performed by one or more programmable processorsexecuting a computer program to perform functions of the invention byoperating on input data and generating output. Method steps can also beperformed by, and apparatus can be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit). Modules can refer to portionsof the computer program and/or the processor/special circuitry thatimplements that functionality.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor receives instructions and data from a read-only memory or arandom access memory or both. A computer comprises a processor forexecuting instructions and one or more memory devices for storinginstructions and data. Generally, a computer also includes, or can beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. Data transmission andinstructions can also occur over a communications network. Informationcarriers suitable for embodying computer program instructions and datainclude all forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in special purposelogic circuitry.

To provide for interaction with a user, the above described techniquescan be implemented on a computing device coupled to or communicatingwith a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer (e.g., interact with a userinterface element). Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback, e.g., visual feedback,auditory feedback, or tactile feedback; and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The above described techniques can be implemented in a distributedcomputing system that includes a back-end component, e.g., as a dataserver, and/or a middleware component, e.g., an application server,and/or a front-end component, e.g., a client computer having a graphicaluser interface and/or a Web browser through which a user can interactwith an example implementation, or any combination of such back-end,middleware, or front-end components. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”),e.g., the Internet, and include both wired and wireless networks.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

Any of the communications, inputs, storage, databases or displaysdiscussed herein may be facilitated through a web site having web pages.The term “web page” as it is used herein is not meant to limit the typeof documents and applications that may be used to interact with theuser. For example, a typical web site may include, in addition tostandard HTML documents, various forms, Java applets, JavaScript, activeserver pages (ASP), common gateway interface scripts (CGI), Flash filesor modules, FLEX, ActionScript, extensible markup language (XML),dynamic HTML, cascading style sheets (CSS), helper applications,plug-ins, and/or the like. A web site, server or computer program mayinclude a web service which includes applications that are capable ofinteracting with other applications over a communications means, such asthe Internet.

Benefits, other advantages, and solutions to problems have beendescribed herein with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any element(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as critical, required, or essentialfeatures or elements of any or all of the claims of the invention.

Systems, methods and computer program products are provided. In thedetailed description herein, references to “various embodiments”, “oneembodiment”, “an embodiment”, “an example embodiment”, “someembodiments,” etc., indicate that the embodiment described may include aparticular feature, structure, or characteristic, but every embodimentmay not necessarily include the particular feature, structure, orcharacteristic. Moreover, such phrases are not necessarily referring tothe same embodiment. Further, when a particular feature, structure, orcharacteristic is described in connection with an embodiment, it issubmitted that it is within the knowledge of one skilled in the art toeffect such feature, structure, or characteristic in connection withother embodiments whether or not explicitly described. After reading thedescription, it will be apparent to one skilled in the relevant art(s)how to implement the disclosure in alternative embodiments.

It should be understood that the detailed description and specificexamples, indicating embodiments of the invention, are given forpurposes of illustration only and not as limitations. Many changes andmodifications within the scope of the invention may be made withoutdeparting from the spirit thereof, and the invention includes all suchmodifications. Corresponding structures, materials, acts, andequivalents of all elements are intended to include any structure,material, or acts for performing the functions in combination with otherelements. Reference to an element in the singular is not intended tomean “one and only one” unless explicitly so stated, but rather “one ormore.” Moreover, when a phrase similar to “one of A, B, or C” or “one ofA, B, and C” is used in the claims or the specification, the phrase isintended to mean any of the following: (1) at least one of A; (2) atleast one of B; (3) at least one of C; (4) at least one of A and atleast one of B; (5) at least one of B and at least one of C; (6) atleast one of A and at least one of C; or (7) at least one of A, at leastone of B, and at least one of C.

What is claimed is:
 1. A data center control and optimization system,comprising: a memory, the memory in communication with a processor, theprocessor when executing a computer program for data center control andoptimization, performs operations comprising: accessing, by theprocessor, input data associated with data center operations, whereinthe input data comprises at least one of inventory data, operationaldata, historical data and external data; determining, by the processor,that the input data does not match a data expectation; forecasting, bythe processor and based upon at least a portion of the input data, toproduce interim data, wherein the interim data and the input data matchthe data expectation; determining, by the processor and based upon atleast one of a data center optimization dimension, the input data andthe interim data, an optimization action; and generating, by theprocessor, an optimization instruction based upon the optimizationaction, wherein executing the optimization instruction causes a changein an operational state associated with the data center operations.
 2. Acomputerized method for controlling and optimizing data centeroperations, comprising: accessing, by a computer, input data associatedwith data center operations, wherein the input data comprises at leastone of inventory data, operational data, historical data and externaldata; determining, by the computer, that the input data does not match adata expectation; forecasting, by the computer and based upon at least aportion of the input data, to produce interim data, wherein the interimdata and the input data match the data expectation; determining, by thecomputer and based upon at least one of a data center optimizationdimension, the input data and the interim data, an optimization action;and generating, by the computer, an optimization instruction based uponthe optimization action, wherein executing the optimization instructioncauses a change in an operational state associated with the data centeroperations.